Globus CLI and SDK version 3 now available!
September 29, 2021 | Stephen Rosen
On March 7th of 2016, a couple of members of the Globus team started working on replacing the globusonline-transfer-api-client package. We debated what to call it. Was it the globus-client? Just globus-python? "What about SDK -- Standard Development Kit?" "Yeah, SDK is good."
The next day, on March 8th, we made the first commit on a second new project, to develop in tandem with the Globus SDK, the Globus CLI. A little over a year after we started work on these projects, we finalized and released version 1.0 of each.
On September 14th, we finalized SDK version 3.0 and today we're announcing Globus CLI version 3.0! I'd like to take a moment to look at what's new, what changed, and where we came from.
A Stable Foundation
Even before we started on the SDK and CLI, we'd learned lessons from maintaining previous packages. While we could often make things easier for users, there was a maintenance cost. The more complex the tools became, the harder it would be to make changes without breaking backwards compatibility for many users. We were careful and constrained. We turned down features we really wanted in the name of maintainability.
Our goal was to follow Semantic Versioning and increment the major version number -- meaning a known-breaking change was made -- very rarely.
After the releases, we weren't certain, but it quickly became clear that we'd made a lot of the right calls. The design we'd chosen, and the restraint we'd shown in avoiding fragile features, proved itself time and again. For over four years, the core of these projects remained unchanged and continued to serve the needs of our community. There was a little blip when we released version 2.0 of each project to drop python2 support. But version 2.0 of the SDK and CLI was really just a bit of cleanup. Other than dropping python2, neither the SDK nor the CLI has made a major, compatibility-breaking change since release.
But now we're here! We're still committed to providing a reliable layer for applications integrating with Globus, but we've had new features and designs we've wanted to get into your hands! It's time for version 3.
What's New?
The SDK version 3 introduces a whole suite of new features, and cleans up many existing ones. We recommend that anyone with an existing SDK application start with our Upgrading Guide to get info on how to handle the breaking changes.
The CLI has gained some new powerful features as well, including profile switching and GCS version 5 support. Let's explore!
Retries and the Transport
If I had to pick just one SDK version 3 feature to talk about, it would be automatic request retries. Under version 2, if a connection error or service error occurred, it would propagate up to the user immediately. Now, under a suite of known conditions, the SDK will automatically sleep for a fraction of a second and retry the request. This is enabled by a new request-sending abstraction (still built on python-requests) called the Transport. You can pass transport_params to any client when instantiating it to parametrize the behaviors of the Transport.
This means that for the most part, SDK users will get more robust code without any additional work.
Goodbye, ~/.globus.cfg ! Hello, tokenstorage!
Both Globus SDK and Globus CLI have ceased to use the config files found at ~/.globus.cfg and /etc/globus.cfg .
In the case of the SDK, we've dramatically simplified configuration. Now, you can tune SDK-level settings with a suite of configuration environment variables. This means your code is less likely to change behavior unexpectedly because of some data in /etc/ .
In the case of the CLI, the change is part of a transition to use a token storage database for logins. The database is placed in ~/.globus/ and allows the CLI to support incremental authorization as new login requirements are discovered. For example, if you run globus collection list $GCSV5_ENDPOINT_ID, you will need to login to the endpoint, and the new storage supports this.
Additionally, the CLI now supports setting an environment variable, GLOBUS_PROFILE. A profile can be named with any string, and switching your profile puts you in a distinct context for login, logout, and session management. If you use multiple distinct Globus accounts, switching your "profile" can allow you to swap between accounts without having to logout and log back in.
All of the new CLI capabilities are enabled by a new component in the SDK, tokenstorage. The underlying mechanism for the Globus CLI is just a StorageAdapter. This makes it possible for SDK users to develop their own flows with functionality similar to the Globus CLI.
Globus Groups and Globus Connect Server v5
Both the SDK and CLI have new features to support the Globus Groups service and new versions of Globus Connect Server v5.
For Groups in the SDK, look at the GroupsClient and the higher-level GroupsManager. Listing groups and adding and removing members are all supported. In the CLI, we've only added globus group list for now, but more commands are on their way.
As for Globus Connect Server v5, we've added support for the GCS Manager API to the SDK, and a suite of new CLI commands -- collection list, show, delete, and update. The SDK support includes some special response-unpacking behavior and other goodies, specifically to make working with the GCS API smoother. We have plans to expand these features as well.
Explicit Pagination
Under SDK version 1 and version 2, we never asked our users to think about which APIs were paginated (returned multiple "pages" of results over the course of several requests) and which ones contained simple arrays. That seemed like a great simplification to us at the time -- users got a "PaginatedResource" object which supported iteration -- but it had some surprising and tricky downsides. It was hard to maintain a single large object which supported many different behaviors, and certain advanced usages were very difficult or even impossible.
Our documentation on paging and paginators explains in greater detail, but the gist of it is that if you use TransferClient.endpoint_search, that now gets one page of results. If you want multiple pages, use TransferClient.paginated.endpoint_search.
A usage with meaning identical to SDK versions 1 and 2 is possible by asking for the items() iterator. But new usage modes, in particular iteration over pages (to get page-level information), are now possible.
Type-Checking with mypy
The Globus SDK now publishes full type annotations for all of its interfaces. We use mypy internally to ensure the quality of the SDK and CLI, but this is fully available to any SDK users.
Type checking can catch a wide range of incorrect usages, as it understands the arguments to functions and classes, and their expected types. Consider this example of incorrect code:
import globus_sdk
tc = globus_sdk.TransferClient(authorrizzer=None)
gm = globus_sdk.GroupsManager(client=tc)authorizer was misspelled, and a GroupsManager consumes a GroupsClient, not a TransferClient. What happens when we run mypy on this?
$ mypy foo.py
foo.py:3: error: Unexpected keyword argument "authorrizzer" for "TransferClient"; did you mean "authorizer"?
foo.py:4: error: Argument "client" to "GroupsManager" has incompatible type "TransferClient"; expected "Optional[GroupsClient]"
Found 2 errors in 1 file (checked 1 source file)mypy was able to find both errors. What's more, it can see that authorrizzer is similar to an acceptable argument name, authorizer, and suggests the fix!
We use the mypy throughout Globus products, and have found that it along with our other testing processes, speeds up development by finding potential bugs earlier. By introducing annotations for the SDK, we've made it possible for everyone to check for a wide class of coding errors with a single command.
For a full introduction to mypy, see the mypy documentation.
ScopeBuilders, globus rename, ErrorInfo, --batch flags, …
The list of features and improvements just keeps coming. We have new scope helpers for constructing Globus Auth scope strings. The globus rename command now takes the endpoint ID only once. We have spruced up error processing with the new ErrorInfo interface. And the --batch flag to globus transfer and globus delete now takes a filename, with - for standard input.
The changes are just too numerous to list them all here. You can browse the SDK Changelog and the CLI Changelog for a full description of all of the changes.
Conclusion
We have many more features we're excited to put into our users' hands in upcoming versions of the SDK and CLI. Keep an eye on our discussion list for news about new versions with the latest features.
For more information, see
You can also file issues and feature requests with us either via support@globus.org , or using the public issue trackers