Tiny Metaprogramming Library

(Difficult-to-grok metaprogramming below. Not for the faint of heart.)

At the recent Urbana-Champaign meeting of the C++ Standardization Committee, Bill Seymour presented his paper N4115: Searching for Types in Parameter Packs which, as its name suggests, describes a library facility for, uh, searching for a type in a parameter pack, among other things. It suggests a template called packer to hold a parameter pack:

// A class template that just holds a parameter pack:
template <class... T> struct packer { };

Many of you are probably already familiar with such a facility, but under a different name:

// A class template that is just a list of types:
template <class... T> struct typelist { };

It became clear in the discussion about N4115 that C++ needs a standard typelist template and some utilities for manipulating them. But what utilities, exactly?

Metaprogramming in the Wild

When it comes to metaprogramming in C++, there is no shortage of prior art. Andrei Alexandrescu started the craze with his Loki library. Boost got in on the act with Boost.MPL, Boost.Fusion, and (currently under development) Hana. All of these libraries are feature-rich and elaborate with their own philosophy, especially Boost.MPL, which takes inspiration from the STL’s containers, iterators, and algorithms.

It wasn’t until recently that I came to doubt MPL’s slavish aping of the STL’s design. The abstractions of the STL were condensed from real algorithms processing real data structures on real computer hardware. But metaprograms don’t run on hardware; they run on compilers. The algorithms and data structures for our metaprograms should be tailored to their peculiar problem domain and execution environment. If we did that exercise, who is to say what abstractions would fall out? Compile-time iterators? Or something else entirely?

Dumb Typelists

If we were to standardize some metaprogramming facilities, what should they look like? It’s an interesting question. N4115 gets one thing right: parameter packs are the compile-time data structure of choice. As of C++11, C++ has language support for lists of types. We would be foolish to work with anything else. IMO, if a standard metaprogramming facility did nothing but manipulate parameter packs — dumb typelists — it would cover 95% of the problem space.

But parameter packs themselves are not first-class citizens of the language. You can’t pass a parameter pack to a function without expanding it, for instance. Wrapping the parameter pack in a variadic typelist template is a no-brainer.

So, like N4115 suggests, this is a sensible starting point:

// A class template that just a list of types:
template <class... T> struct typelist { };

It’s a rather inauspicious start, though; clearly we need more. But what? In order to answer that, we need to look at examples of real-world metaprogramming. With concrete examples, we can answer the question, What he heck is this stuff good for, anyway? And for examples, we have to look no farther than the standard library itself.

Tuple_cat

Stephan T. Lavavej drew my attention to the tuple_cat function in the standard library, a function that takes N tuples and glues them together into one. It sounds easy, but it’s tricky to code efficiently, and it turns out to be a great motivating example for metaprogramming facilities. Let’s code it up, and posit a few typelist algorithms to make our job easier. (All the code described here can be found in my range-v3 library on GitHub.)

First, I’m going to present the final solution so you have an idea of what we’re working toward. Hopefully, by the time you make it to the end of this post, this will make some sort of sense.

namespace detail
{
    template<typename Ret, typename...Is, typename ...Ks,
        typename Tuples>
    Ret tuple_cat_(typelist<Is...>, typelist<Ks...>,
        Tuples tpls)
    {
        return Ret{std::get<Ks::value>(
            std::get<Is::value>(tpls))...};
    }
}

template<typename...Tuples,
    typename Res =
        typelist_apply_t<
            meta_quote<std::tuple>,
            typelist_cat_t<typelist<as_typelist_t<Tuples>...> > > >
Res tuple_cat(Tuples &&... tpls)
{
    static constexpr std::size_t N = sizeof...(Tuples);
    // E.g. [0,0,0,2,2,2,3,3]
    using inner =
        typelist_cat_t<
            typelist_transform_t<
                typelist<as_typelist_t<Tuples>...>,
                typelist_transform_t<
                    as_typelist_t<make_index_sequence<N> >,
                    meta_quote<meta_always> >,
                meta_quote<typelist_transform_t> > >;
    // E.g. [0,1,2,0,1,2,0,1]
    using outer =
        typelist_cat_t<
            typelist_transform_t<
                typelist<as_typelist_t<Tuples>...>,
                meta_compose<
                    meta_quote<as_typelist_t>,
                    meta_quote_i<std::size_t, make_index_sequence>,
                    meta_quote<typelist_size_t> > > >;
    return detail::tuple_cat_<Res>(
        inner{},
        outer{},
        std::forward_as_tuple(std::forward<Tuples>(tpls)...));
}

That’s only 43 lines of code. The implementation in stdlib++ is 3x longer, no easier to understand (IMHO), and less efficient. There’s real value in this stuff. Really.

Let’s look first at the return type:

// What return type???
template< typename ...Tuples >
???? tuple_cat( Tuples &&... tpls );

You can think of a tuple as a list of types and a list of values. To compute the return type, we only need the list of types. So a template that turns a tuple into a typelist would be useful. Let’s call it as_typelist. It takes a tuple and does the obvious. (Another possibility would be to make tuples usable as typelists, but let’s go with this for now.)

If we convert all the tuples into typelists, we end up with a list of typelists. Now, we want to concatenate them. Ah! We need an algorithm for that. Let’s call it typelist_cat in honor of tuple_cat. ^{_{(Functional programmers: typelist_cat is join in the List Monad. Shhh!! Pass it on.)}} Here’s what we have so far:

// Concatenate a typelist of typelists
template< typename ...Tuples >
typelist_cat_t<
    typelist< as_typelist_t< Tuples >... >
>
tuple_cat( Tuples &&... tpls );

Here, I’m following the convention in C++14 that some_trait_t<X> is a template alias for typename some_trait<X>::type.

The above signature isn’t right yet — tuple_cat needs to return a tuple, not a typelist. We need a way to convert a typelist back to a tuple. It turns out that expanding a typelist into a variadic template is a useful operation, so let’s create an algorithm for it. What should it be called? Expanding a typelist into a template is a lot like expanding a tuple into a function call. There’s a tuple algorithm for that in the Library Fundamentals TS called apply. So let’s call our metafunction typelist_apply. It’s implementation is short and interesting, so I’ll show it here:

template<template<typename...> class C, typename List>
struct typelist_apply;

template<template<typename...> class C, typename...List>
struct typelist_apply<C, typelist<List...>>
{
    using type = C<List...>;
};

The first parameter is a rarely-seen template template parameter. We’ll tweak this interface before we’re done, but this is good enough for now.

We can now write the signature of tuple_cat as:

template<typename...Tuples>
typelist_apply_t<
    std::tuple,
    typelist_cat_t<typelist<as_typelist_t<Tuples>...> > >
tuple_cat(Tuples &&... tpls);

Not bad, and we have discovered three typelist algorithms already.

Tuple_cat Implementation

It’s time to implement tuple_cat, and here’s where things get weird. It’s possible to implement it by peeling off the first tuple and exploding it into the tail of a recursive call. Once you’ve recursed over all the tuples in the argument list, you have exploded all the tuple elements into function arguments. From there, you bundle them into a final tuple and you’re done.

That’s a lot of parameter passing.

Stephan T. Lavavej tipped me off to a better way: Take all the tuples and bundle them up into a tuple-of-tuples with std::forward_as_tuple. Since tuples are random-access, a tuple of tuples is like a jagged 2-dimensional array of elements. We can index into this 2-dimensional array with (i,j) coordinates, and if we have the right list of (i,j) pairs, then we can fetch each element in turn and build the resulting tuple in one shot, without all the explosions.

To make this more concrete, imaging the following call to tuple_cat:

std::tuple<int, short, long> t1;
std::tuple<> t2;
std::tuple<float, double, long double> t3;
std::tuple<void*, char*> t4;

auto res = tuple_cat(t1,t2,t3,t4);

We want the result to be a monster tuple of type:

std::tuple<int, short, long, float, double,
           long double, void*, char*>

This call to tuple_cat corresponds to the following list of (i,j) coordinates:

[(0,0),(0,1),(0,2),(2,0),(2,1),(2,2),(3,0),(3,1)]

Below is a tuple_cat_ helper function that takes the i‘s, j‘s, and tuple of tuples, and builds the resulting tuple:

template<typename Ret, typename...Is, typename ...Js,
    typename Tuples>
Ret tuple_cat_(typelist<Is...>, typelist<Js...>,
    Tuples tpls)
{
    return Ret{std::get<Js::value>(
        std::get<Is::value>(tpls))...};
}

Here, the Is and Js are instances of std::integral_constant. Is contains the sequence [0,0,0,2,2,2,3,3] and Js contains [0,1,2,0,1,2,0,1].

Well and good, but how to compute Is and Js? Hang on tight, because Kansas is going bye bye.

Higher-Order Metaprogramming, Take 1

Let’s first consider the sequence of Js since that’s a little easier. Our job is to turn a list of typelists [[int,short,long],[],[float,double,long double],[void*,char*]] into a list of integers [0,1,2,0,1,2,0,1]. We can do it in four stages:

1. Transform the lists of typelist into a list of typelist sizes: [3,0,3,2],
2. Transform that to a list of index sequences [[0,1,2],[],[0,1,2],[0,1]] using std::make_index_sequence,
3. Transform the std::index_sequence into a typelist of std::integral_constants with as_typelist, and
4. Flatten that into the final list using typelist_cat.

By now it’s obvious that we’ve discovered our fourth typelist algorithm: typelist_transform. Like std::transform, typelist_transform takes a sequence and a function, and returns a new sequence where each element has been transformed by the function. ^{_{(Functional programmers: it’s fmap in the List Functor)}}. Here’s one possible implementation:

template<typename List, template<class> class Fun>
struct typelist_transform;

template<typename ...List, template<class> class Fun>
struct typelist_transform<typelist<List...>, Fun>
{
    using type = typelist<Fun<List>...>;
};

Simple enough.

Metafunction Composition

Above, we suggested three consecutive passes with typelist_transform. We can do this all in one pass if we compose the three metafunctions into one. Metafunction composition seems like a very important utility, and it’s not specific to typelist manipulation. So far, we’ve been using template template parameters to pass metafunctions to other metafunctions. What does metafunction composition look like in that world? Below is a higher-order metafunction called meta_compose that composes two other metafunctions:

template<template<class> class F0,
         template<class> class F1>
struct meta_compose
{
    template<class T>
    using apply = F0<F1<T>>;
};

Composing two metafunction has to result in a new metafunction. We have to use an idiom to “return” a template by defining a nested template alias apply which does the composition.

Seems simple enough, but in practice, this quickly becomes unwieldy. If you want to compose three metafunctions, the code looks like:

meta_compose<F0, meta_compose<F1, F2>::template apply>
    ::template apply

Gross. What’s worse, it’s not very general. We want to compose std::make_index_sequence, and that metafunction doesn’t take a type; it takes an integer. We can’t pass it to to a meta_compose. Let’s back up.

Higher-Order Metaprogramming, Take 2

What if, instead of passing meta_compose<X,Y>::template apply to a higher-order function like typelist_transform, we just passed meta_compose<X,Y> and let typelist_transform call the nested apply? Now, higher-order functions like typelist_transform take ordinary types instead of template template parameters. typelist_transform would now look like:

template<typename ...List, typename Fun>
struct typelist_transform<typelist<List...>, Fun>
{
    using type =
        typelist<typename Fun::template apply<List>...>;
};

That complicates the implementation of typelist_transform, but makes the interface much nicer to deal with. The concept of a class type that behaves like a metafunction comes from Boost.MPL, which calls it a Metafunction Class.

We can make Metafunction Classes easier to deal with with a little helper that applies the nested metafunction to a set of arguments:

template<typename F, typename...As>
using meta_apply = typename F::template apply<As...>;

With meta_apply, we can rewrite typelist_transform as:

template<typename ...List, typename Fun>
struct typelist_transform<typelist<List...>, Fun>
{
    using type = typelist<meta_apply<Fun, List>...>;
};

That’s not bad at all. Now we can change meta_compose to also operate on Metafunction Classes:

template<typename F1, typename F2>
struct meta_compose
{
    template<class T>
    using apply = meta_apply<F1, meta_apply<F2, T>>;
};

With a little more work, we could even make it accept an arbitrary number of Metafunction Classes and compose them all. It’s a fun exercise; give it a shot.

Lastly, now that we have Metafunction Classes, we should change typelist_apply to take a Metafunction Class instead of a template template parameter:

template<typename C, typename...List>
struct typelist_apply<C, typelist<List...> >
{
    using type = meta_apply<C, List...>;
};

Metafunctions to Metafunction Classes

Recall the four steps we’re trying to evaluate:

1. Transform the lists of typelist into a list of typelist sizes: [3,0,3,2],
2. Transform that to a list of index sequences [[0,1,2],[],[0,1,2],[0,1]] using std::make_index_sequence,
3. Transform the std::index_sequence into a typelist of std::integral_constants with as_typelist, and
4. Flatten that into the final list using typelist_cat.

In step (1) we get the typelist sizes, so we need another typelist algorithm called typelist_size that fetches the size of type typelist:

template<typename...List>
struct typelist_size<typelist<List...> >
  : std::integral_constant<std::size_t, sizeof...(List)>
{};

We’re going to want to pass this to meta_compose, but typelist_size is a template, and meta_compose is expecting a Metafunction Class. We can write a wrapper:

struct typelist_size_wrapper
{
    template<typename List>
    using apply = typelist_size<List>;
};

Writing these wrappers is quickly going to get tedious. But we don’t have to. Below is a simple utility for turning a boring old metafunction into a Metafunction Class:

template<template<class...> class F>
struct meta_quote
{
    template<typename...Ts>
    using apply = F<Ts...>;
};

The name quote comes from LISP via Boost.MPL. With meta_quote we can turn the typelist_size template into a Metafunction Class with meta_quote<typelist_size>. Now we can pass it to either meta_compose or typelist_transform.

Our steps call for composing three metafunctions. It will look something like this:

meta_compose<
    meta_quote<as_typelist_t>,            // Step 3
    meta_quote<std::make_index_sequence>, // Step 2
    meta_quote<typelist_size_t> >         // Step 1

As I already mentioned, std::make_index_sequence takes an integer not a type, so it can’t be passed to meta_quote. This is a bummer. We can work around the problem with a variant of meta_quote that handles those kinds of templates. Let’s call it meta_quote_i:

template<typename Int, template<Int...> class F>
struct meta_quote_i
{
    template<typename...Ts>
    using apply = F<Ts::value...>;
};

With meta_quote_i, we can compose the three functions with:

meta_compose<
    meta_quote<as_typelist_t>,              // Step 3
    meta_quote_i<std::size_t,
                 std::make_index_sequence>, // Step 2
    meta_quote<typelist_size_t> >           // Step 1

Now we can pass the composed function to typelist_transform:

typelist_transform_t<
    typelist<as_typelist_t<Tuples>...>,
    meta_compose<
        meta_quote<as_typelist_t>,
        meta_quote_i<std::size_t, make_index_sequence>,
        meta_quote<typelist_size_t> > > >;

Voila! We have turned our lists of tuples into the list of lists: [[0,1,2],[],[0,1,2],[1,2]]. To get the final result, we smoosh this into one list using typelist_cat:

// E.g. [0,1,2,0,1,2,0,1]
typelist_cat_t<
    typelist_transform_t<
        typelist<as_typelist_t<Tuples>...>,
        meta_compose<
            meta_quote<as_typelist_t>,
            meta_quote_i<std::size_t, make_index_sequence>,
            meta_quote<typelist_size_t> > > >;

The result is the K indices that we pass to the tuple_cat_ helper. And to repeat from above, the I indices are computed with:

// E.g. [0,0,0,2,2,2,3,3]
typelist_cat_t<
    typelist_transform_t<
        typelist<as_typelist_t<Tuples>...>,
        typelist_transform_t<
            as_typelist_t<make_index_sequence<N> >,
            meta_quote<meta_always> >,
        meta_quote<typelist_transform_t> > >;

I won’t step through it, but I’ll draw your attention to two things: on line (7) we make use of a strange type called meta_always (described below), and on line (8) we pass typelist_transform as the function argument to another call of typelist_transform. Talk about composability!

So what is meta_always? Simply, it’s a Metafunction Class that always evaluates to the same type. Its implementation couldn’t be siimpler:

template<typename T>
struct meta_always
{
    template<typename...>
    using apply = T;
};

I’ll leave you guys to puzzle out why the above code works.

Summary

I set out trying to find a minimal useful set of primitives for manipulating lists of types that would be fit for standardization. I’m happy with the result. What I’ve found is that in addition to the typelist template, we need a small set of algorithms like the ones needed to implement tuple_cat:

• typelist_apply
• typelist_size
• typelist_transform
• typelist_cat
• as_typelist

Some other typelist algorithms come up in other metaprogramming tasks:

• make_typelist (from a count and type)
• typelist_push_front
• typelist_push_back
• typelist_element (indexing into a typelist)
• typelist_find and typelist_find_if
• typelist_foldl (aka, accumulate) and typelist_foldr
• etc.

In addition, for the sake of higher-order metafunctions like typelist_transform and typelist_find_if, it’s helpful to have a notion of a Metafunction Class: an ordinary class type that can be used as a metafunction. A small set of utilities for creating and manipulating Metafunction Classes is essential for the typelist algorithms to be usable:

• meta_apply
• meta_quote
• meta_quote_i
• meta_compose
• meta_always

For other problems, the ability to partially apply (aka bind) Metafunction Classes comes in very handy:

• meta_bind_front
• meta_bind_back

And that’s it, really. In my opinion, those utilities would meet the needs of 95% of all metaprograms. They are simple, orthogonal, and compose in powerful ways. Since we restricted ourselves to the typelist data structure, we ended up with a design that is vastly simpler than Boost.MPL. No iterators needed here, which makes sense since iterators are a pretty stateful, iterative abstraction, and metaprogramming is purely functional.

One Last Thing…

Below is one more metafunction to tickle your noodle. It’s an N-way variant of transform: it takes a list of typelists and a Metafunction Class, and builds a new typelist by mapping over all of them. I’m not suggesting this is important or useful enough to be in the standard. I’m only showing it because it demonstrates how well these primitive operations compose to build richer functionality.

// ([[a,b,c],[x,y,z]], F) -> [F(a,x),F(b,y),F(c,z)]
template<typename ListOfLists, typename Fun>
struct typelist_transform_nary :
  typelist_transform<
    typelist_foldl_t<
      ListOfLists,
      make_typelist<
        typelist_front_t<ListOfLists>::size(),
        Fun>,
      meta_bind_back<
        meta_quote<typelist_transform_t>,
        meta_quote<meta_bind_front> > >,
    meta_quote<meta_apply> >
{};

Enjoy!

Update: This comment by tkamin helped me realize that the above typelist_transform_nary is really just the zipWith algorithm from the functional programming world. I’ve renamed it in my latest code, and provided a typelist_zip metafunction that dispatches to typelist_zip_with with meta_quote<typelist> as the function argument. Very nice!