[NEW] devel/awscli2: Universal Command Line Interface for Amazon Web Services V2

[hsw_bitmark.com]

Summary

[NEW] devel/awscli2: Universal Command Line Interface for Amazon Web Services V2

This package provides a unified command line interface to many Amazon Web
Services.

The currently supported services include:

* Amazon Elastic Compute Cloud (Amazon EC2)
* Elastic Load Balancing
* Auto Scaling
* AWS CloudFormation
* AWS Elastic Beanstalk
* Amazon Simple Notification Service (Amazon SNS)
* Amazon Simple Queue Service (Amazon SQS)
* Amazon Relational Database Service (Amazon RDS)

Reviewed_by: diizzy, lwhsu, koobs
Differential_Revision: D35506

This includes both the CLI and its associated dependency devel/py-awscrt

The V1 port devel/awscli has added CONFLICTS since the V1 and V2 ports
install files of the same name

Test Plan

• portlint: ??
• testport: poudriere bulk -j 13_0-amd64 devel/py-awscrt devel/awscli devel/awscli2
• maketest: ??

[koobs]

devel/awscli2/Makefile

2

There doesn't appear to be a PyPI package registered with this name (awscli2)

Whats the nature of the distinction and difference between awscli and awscli2?

Depending on the answer, it may be a better option to rename awscli -> awscli1 leaving awscli for a major version upgrade (to 2), but i see https://pypi.org/project/awscli/ is still v1.x

Do we know what official name the v2 branch will use in PyPI?

4

Add python as a secondary (virtual) CATEGORIES

24

Match USES=python:<version-spec> as closely as possible to upstream version support declarations as possible.

Upstream declares only 3.8/3.9 support [1]. If upstream tests against and the test suite passes with more (or later) Python version, ask upstream to adjust python_Requires accordingly, and cap <version-spec> to currently supported and tested versions

[1] https://github.com/aws/aws-cli/blob/2.7.9/setup.cfg#L17=

25

Is this noflavors just matching the existing declaration of same in awscli?

28

Open issue asking upstream to register/ship a PyPI sdist if they haven't already, and use that if it exists, otherwise USE_GITHUB is fine in the short term

[hsw_bitmark.com]

• Fix Makefiles
• Fix building of awscli2 autocompletion index

[hsw_bitmark.com]

my understanding is:

• that V1.x support will continue and be available PyPi
• V2.x will not be available on PyPI but via a binary download from Amazon Currently only Mac/Win/Linux Installers are available
• V2.x uses different auth API which is why I chose awscli2 to avoid confusion if someone upgraded an existing awscli then they would be get authentication failures
• the noflavors was copied. I though this meant just one package using the default Python version so upgrades automatically each time a new Python is made as default. unlike Ansible which has to be deleted and reinstalled each time Python changes

Would you look at my revisions and let me know where improvements can be made.
Thanks

[lwhsu]

devel/awscli/Makefile

23 ↗

(On Diff #107157)

In bsd.port.mk;

# CONFLICTS     - A list of package name patterns that the port conflicts
#                 with, separated by blanks.  The names may include shell
#                 pattern meta-characters "*", "?", "[", "]", and "!".
#                 Example: apache*-1.2* apache*-1.3.[012345] apache-*+ssl_*

So this might not work, or CONFLICTS does support category/portname pattern now?

[hsw_bitmark.com]

• remove category from CONFLICTS, set minimum Python version

[koobs]

In D35506#805709, @hsw_bitmark.com wrote:

my understanding is:

• that V1.x support will continue and be available PyPi
• V2.x will not be available on PyPI but via a binary download from Amazon Currently only Mac/Win/Linux Installers are available
• V2.x uses different auth API which is why I chose awscli2 to avoid confusion if someone upgraded an existing awscli then they would be get authentication failures
• the noflavors was copied. I though this meant just one package using the default Python version so upgrades automatically each time a new Python is made as default. unlike Ansible which has to be deleted and reinstalled each time Python changes

Would you look at my revisions and let me know where improvements can be made.
Thanks

Thanks for the detail.

Keeping awscli matching the registered PyPI package and name is fine then.

We can switch it over if and when upstream replaces awscli with the 2.x version (or follow whatever it is that they do)

devel/py-awscrt/files/patch-crt_s2n_utils_s2n__fork__detection.c

7 ↗

(On Diff #107203)

Does this resolve a particular issue/warning/function?

Please add a comment block with summary reason for the patch, and include upstream references (issues, pr's, commits) links where they exist

devel/py-awscrt/files/patch-setup.py

6 ↗

(On Diff #107203)

This patch looks to be a noop?

But while we're here, if -Werror is included in the default compiler invocations, please, either:

2. Add CFLAGS+-Wno-error to override it, OR
3. If that doesnt work due to argument order, patch it out.

[koobs]

• the noflavors was copied. I though this meant just one package using the default Python version so upgrades automatically each time a new Python is made as default. unlike Ansible which has to be deleted and reinstalled each time Python changes

You correctly describe the behaviour when PKGNAMEPREFIXES dont exist (like this port) and Python default versions change, but there's additional nuance:

• Official package builders only produce a single version of Python ports (using the current default versions) at the moment, but that is a function of there just being a policy not to, and is not a function of noflavor being defined or not. If that policy changed tomorrow, all ports without noflavors would automatically do the right thing and have packaged produced for them, as expected.
• As mentioned in the original reply, noflavors also switches the default expectation of users to be able to choose what Python version they want a package or set of packages to use, a principle of least astonishment (POLA) violation, prevents their ability to choose when to change/upgrade Python versions for their tooling/packaging, and is contrary to the default behaviour and value Python has sought to create for Python on FreeBSD for users.
• noflavors was technically created only as an escape hatch for automatic Ports (not Python specifically) flavour creation. It just so happened that Python ended up being a first and major flavor consumer, so the perception is its a python port thing.

In terms of what we do for these ports now, we have the following to consider:

• noflavors is already defined in awscli, but its existence in that port is not a warrant itself that awscli2 must follow.

• the package name has changed in the past (with and without PYTHON_PGNAMEPREFIX), and removing noflavors from awscli would mean changing it again:

2018-01-02

Affects: users of devel/awscli

Author: [e-mail]

Reason: 
  The package name for devel/awscli has changed (back) to just awscli.
  The introduction of flavors erroneously caused the awscli package to
  be renamed to py27-awscli.

I would argue the original addition of noflavors was incorrect to begin with, contrary to how the UPDATING message above sounds (that having the prefix was erroneous)

• As this review is for the addition of a new/separate port, there's less of a need to gate or couple the commit of this port, to changes in another port, but again, not removing nofvlaors from awscli isnt itself warrant to include it in awscli2.

• If awscli was landing today with review, noflavors would be questioned and likely not included. With that in mind, its likely in the long-term state for noflavors to be removed from awscli (along with any other port that does that, for that reason as well as the "this is how we package python ports in all but exceptional cases" and "user choice" grounds.

devel/awscli2/Makefile

13–24

• Are these really build time depends? Python packages usually declare build dependencies as setup_requires. Upstream only has these defined for RUN_DEPENDS (install_requires):

install_requires = [
    'botocore==1.27.14',
    'docutils>=0.10,<0.17',
    's3transfer>=0.6.0,<0.7.0',
    'PyYAML>=3.10,<5.5',
    'colorama>=0.2.5,<0.4.5',
    'rsa>=3.1.2,<4.8',
]

Could you explain the rationale/need for dependencies not declared by upstream?

25–36

Upstream only has these defined for RUN_DEPENDS (install_requires):

install_requires = [
    'botocore==1.27.14',
    'docutils>=0.10,<0.17',
    's3transfer>=0.6.0,<0.7.0',
    'PyYAML>=3.10,<5.5',
    'colorama>=0.2.5,<0.4.5',
    'rsa>=3.1.2,<4.8',
]

1. Could you explain the rationale/need for dependencies not declared by upstream?
2. Upstream also pins max versions. Python ports must match declared dependencies as closely/exactly as possible. This means including <X in *_DEPENDS declarations like:

	${PYTHON_PKGNAMEPREFIX}colorama>=0.2.5,<0.4.5:devel/py-colorama@${PY_FLAVOR} \

[koobs]

Apart from inline review items to address:

• If upstream provides test suites with these packages, please add TEST_DEPENDS and a (do-)test: target for test dependencies and to run the test command respectively
• Confirm both ports pass portlint/portfmt/portclippy (whatever)
• Confirm both ports passes poudriere for a default and non default Python version. The poudriere mention in the existing TEST PLAN didnt mention which Python versions. If that was the default (mention which version), and confirm poudriere passes for a non default version too

Thanks so much for all your hard work!

[hsw_bitmark.com]

• devel/awscli2: New port: AWS CLI version 2.x + dependencies
• Fix Makefiles
• remove category from CONFLICTS, set minimum Python version
• [WIP] update versions, align patches, do-test target

[hsw_bitmark.com]

I have updated the Makefiles and patches to latest versions.

The awscli make test is failing because there is a fresh download on an unpatched py-aws-crt during the test build
this causes compilation to fail because __BSD_VISIBLE is not defined adding this to TEST_ENV had no effect.

Since the main branch of https://github.com/aws/s2n-tls contains the required fixes but ttps://github.com/awslabs/aws-crt-python
does not have an updated s2n submodule, I am not sure what to do:

• Just remove the test code
• Or, comment it out with a message to retest when 0.13.14 of the crt is released

I would appreciate any advice on how to proceed with this port.

devel/py-awscrt/files/patch-setup.py

6 ↗

(On Diff #107203)

it changes fatal_warnings → fatal-warnings maybe gcc can accept either, but I clang did not

[hsw_bitmark.com]

bring up todate and comment out test target since this cannot work until
devel/py-awscrt internal s2n submodule is updated to incle FreeBSD patches

[hsw_bitmark.com]

remove py-awscrt patch file

[hsw_bitmark.com]

update with latest version, devel/py-awscrt has been ported by someone else, so remove those files from this revision,
except for the missing LDFLAGS in the devel/py-awscrt/Makefile

[hsw_bitmark.com]

for make test I tried patching the tox.ini with a setenv= to set CFLAGS and LDFLAGS to build values suitable for the py-awscrt "wheel" but it fails because of the bug in its setup.py.

The failure occurs during:

* Installing packages in isolated environment... (awscrt>=0.12.4,<=0.13.15, colorama>=0.2.5,<0.4.5, cryptography>=3.3.2,<37.0.0, distro>=1.5.0,<1.7.1, docutils>=0.10,<0.17.2, jmespath>=0.7.1,<1.1.0, prompt-toolkit>=3.0.24,<3.1.0, python-dateutil>=2.1,<3.0.0, ruamel.yaml>=0.15.0,<=0.17.21, urllib3>=1.25.4,<1.27, wcwidth<0.2.0, wheel)

the error being:

ld: error: unknown argument '-fatal_warnings', did you mean '-fatal-warnings'

[hsw_bitmark.com]

just update to keep track of latest versions

still waiting on devel/py-awscrt to have upstream updates:

• has its s2n submodule is updated to include FreeBSD fixes
• had '-fatal_warnings' changed to '-fatal-warnings'
• has '-lexecinfo' added to the linker options
• see: https://github.com/awslabs/aws-crt-python/pull/376

[hsw_bitmark.com]

just update dependencies

[hsw_bitmark.com]

remove som extraneous files

[arrowd]

@hsw_bitmark.com Can you please comment/resolve remaining comments?

[hsw_bitmark.com]

I think I have addressed the current Issues, but there more recent versions the most recent is 2.9.22
my port is still on 2.7 branch and I wonder if I should try 2.9 branch to see if that workable first

devel/awscli2/Makefile

2

My understanding is that awscli will continue to be supported via PyPi, but awscli2 will not

28

https://pypi.org/project/awscliv2/ is the unofficial port for V2 on PyPi, but it needs Docker since it installs a docker image. As far as I can tel this port just installs an installer to fetch binaries from AWS, which supports Windows/MacOS/ and 64-bit versions of recent distributions of CentOS, Fedora, Ubuntu, Amazon Linux 1, Amazon Linux 2 and Linux ARM. No mention of BSD support I could find. This is why I made this port direcly compiled from github

[hsw_bitmark.com]

I have looked at more recent versions up to 2.11.6 and the build system is completely different, basically ./configure + make
and it uses pyinstaller to 5.8 to do the build.

devel/py-pyinstaller is currently PyInstaller-3.5.tar.gz and its PyPi distfile is now all lowercase and appears to have had all the docs remove
so this port would need to be fixed first. Latest dist is: pyinstaller-5.9.tar.gz

Currently:

1. Updated devel/py-pyinstaller to 5.8 and remove all doc build from its Makefile (most other py-* do not seem to have docs)
2. Add new port devel/py-pyinstaller-hooks-contrib (required by awscli 2.11.6)
3. Update devel/py-awscrt to 0.16.13 and patch missing link info
4. Update devel/awscli2 to 2.11.6

The result which is a stand-alone pyinstller system, i.e., all deps in /usr/local/lib/aws-cli

[hsw_bitmark.com]

recreate port based on 2.11.6 which is a no deps pyinstaller type of package

[hsw_bitmark.com]

fix error during configure because of bash specific code

[hsw_bitmark.com]

move WWW from pkg-descr to Makefile

[hsw_bitmark.com]

simplify pkg-descr

[hsw_bitmark.com]

The port has been simplified, and most of the original patched have been removed

• configure need #! fix to use bash since it uses a bash specific read -r -d '' to read multiple lines to a shell variable, FreeBSD /bin/sh does not support this
• since FreeBSD Python is be default built without embedded sqlite3, the py-sqlite3 is an extra dependency for build/run
• all other Python packages are built into the port by PyInstaller

[hsw_bitmark.com]

update to highest version currently available 2.13.23
this needs Python v3.10 or later

[hsw_bitmark.com]

update to latest source revision: 2.15.8

[arrowd]

devel/awscli2/Makefile

41

What is USE_MAKE?

47

Aren't these files come from ${WRKSRC}/bin/ if yes, we can do USES= shebangfix.

[hsw_bitmark.com]

devel/awscli2/Makefile

41

The USE_MAKE seems to be just me misremembering something. I will remove this.

47

If I remove those lines, and my REINPLACE_CMD line 55 below, add shebangfix to the USES= I get the following errors with poudriere testport:

====> Running Q/A tests (stage-qa)
Error: '/wrkdirs/usr/ports/devel/awscli2/work/stage/usr/local/lib/aws-cli/bin/python' is an invalid shebang you need USES=shebangfix for 'lib/aws-cli/bin/aws'
Error: '/wrkdirs/usr/ports/devel/awscli2/work/stage/usr/local/lib/aws-cli/bin/python' is an invalid shebang you need USES=shebangfix for 'lib/aws-cli/bin/aws_completer'
Error: 'lib/aws-cli/bin/aws' is referring to /wrkdirs/usr/ports/devel/awscli2/work/stage
Error: 'lib/aws-cli/bin/aws_completer' is referring to /wrkdirs/usr/ports/devel/awscli2/work/stage

the #! line looks like #!/wrkdirs/usr/ports/devel/awscli2/work/stage/usr/local/lib/aws-cli/bin/python
so I think it is not detected as a valid Python interpreter.

[hsw_bitmark.com]

remove USE_MAKE and update to latest patch 2.15.17