2017 Perl Toolchain Summit

Tags:

This year I had one goal for CPAN Testers: Replace the current Metabase API with a new API that did not write to Amazon SimpleDB. The current high-availability database that raw incoming test reports are written is Amazon SimpleDB behind an API called Metabase. Metabase is a highly-flexible data storage API designed to work with massive, unstructured data sets and still allow for sane organization and storage of data. Unfortunately, Amazon SimpleDB is as it says on the tin: Simple. Worse, it's expensive: Like most Amazon services, it charges for usage, so there's a huge incentive for CPAN Testers to use it as little as possible (which made some of the code quite obtuse).

So, I made a plan to excise the Metabase. Since we already cached every raw test report locally in the CPAN Testers MySQL database, I planned to write a new Metabase API that wrote directly to the cache, and then adjust the backend processing to read from the cache. I spent the better part of a month working through all the Metabase APIs, how the data was stored in the database, and how to translate between a simple JSON format and the serialized Metabase objects. However, some proper schema design prevented me from finishing this project: A single NOT NULL column could not be changed to allow nulls very easily, it being a 600GB table. The one time where a well-designed schema was a bad thing!

But then Garu, author of cpanm-reporter and CPAN::Testers::Common::Client came up with an idea to make a new test report format. These new reports would have to be stored in a new place, and I discovered that MySQL had recently started building some rich JSON tooling. Making a new JSON test report format and storing it in our new high-availability MySQL cluster seemed like a perfect solution for storing our raw test reports.

After a few weeks of discussion, I finally realized that it would be an easier task to make a backwards-compatible Metabase API write to the new test report MySQL table, even though it increased the amount of work that needed to be done:

  • Complete the new test report format schema (Garu)
  • Write the new backwards-compatibility Metabase API (Me)
  • Write a new test report processor that writes to the old Metabase cache tables (Joel Berger)
  • Write a migration script from the old Metabase cache tables to the new test report JSON object (?)

With that plan, I headed for Lyon.

Continue reading 2017 Perl Toolchain Summit...

CPAN Testers Has a New API

Tags:

As part of the MetaCPAN hackathon, meta::hack, I was invited to work on the CPAN Testers integration. CPAN Testers is a community of CPAN users who send in test reports for CPAN modules as they are uploaded. MetaCPAN adds a summary of those test reports to every CPAN distribution to help you determine which module you'd most like to use. For quite a few months, this integration was broken, and the nature of the current integration (a SQLite database) means it is not as generally useful as it could be.

So, I decided that the best way to improve the CPAN Testers / MetaCPAN integration was to build a new CPAN Testers API. This API uses the CPAN Testers schema to expose CPAN Testers data using a JSON API. This API is built using the Mojolicious web framework, and an OpenAPI specification (using Mojolicious::Plugin::OpenAPI.

Continue reading CPAN Testers Has a New API...

meta::hack log

Tags:

Last week, I attended meta::hack, the MetaCPAN hackathon in Chicago. I'm the maintainer for CPAN Testers, the central database for CPAN users to send in test reports on CPAN distributions and one of MetaCPAN's data sources. I asked to join them so I could improve how MetaCPAN consumes CPAN Testers data, and ensure the stability and reliability of that consumption.

Here's a detailed log of what I was able to accomplish, and information on the new development of CPAN Testers.

Continue reading meta::hack log...

Perl QA Hackathon - CPANTesters

Tags:

This year, I was invited to the Perl QA Hackathon in Rugby, UK. It was wonderful to meet all the Perl people I'd been interacting with all this time.

My goals going into the hackathon weren't that clear: I've recently begun adopting the CPANTesters project, and I had to take the opportunity to talk with its former leader, Barbie, fix some current issues, and then...

While Barbie fixed the version summaries and Metacpan issue, I started work on an automated deploy for CPANTesters using Rex, which will allow for reproducible deployments and development virtual machines, and I began keeping track of the project and future goals in a CPANTesters project meta-repository, which should help with keeping CPANTesters going as an open community project. I'll be making future blog posts on both of these, though I've spoken about Rex before.

Thanks to Barbie for 10 years of CPANTesters, and special thanks to Capside for their donation, both monetary and avian, as they sent Oriol Soriano to help with some CPANTesters tasks.

And finally, thanks to all the other sponsors of the hackathon. Without their support, we couldn't do all the work we do on the Perl ecosystem.