I'm having hard time imagining how can I change get*DatasetType to also return extra info from configuration object without making it overly complicated and keeping it future-proof.

How about adding extra method to PipelineTask which returns the value of "scalar" for given field with default implementation that sub-classes can override as well, e.g.:

    @classmethod

    def getDatasetIsScalar(cls, config, key):

        field = config.get(key)

        if isinstance(field, (InputDatasetConfig, OutputDatasetConfig)):

            return field.scalar

        return False

I do not particularly like this, if we need to add something else to configuration it would mean new method like this will be needed. Should we just rename get*DatasetType() to get*DatasetConfig() and return configuration objects from those (and let sub-classes override those)?

Should we just rename get*DatasetType() to get*DatasetConfig() and return configuration objects from those (and let sub-classes override those)?

I think this is a bit closer to what we want.  I'm a bit worried it may be confusing to use a Config class as the return element type of these methods, despite the fact that they're likely to have all of the attributes we need*.*

Should we invent a new class for "DatasetType as used by a PipelineTask" that could be returned instead?  Those could have a DatasetType instance, and a way to construct from *DatasetConfig (taking over the logic in PipelineTask.makeDatasetType), and a scalar attribute, and whatever else we need them to have in the future.  We could start with a simple namedtuple for now.

I was trying to avoid introducing new class (semantically it should be ~equivalent to *DatasetConfig classes) but it may be the safest option for the future extensions. I'll try to implement something not too complicated.

Thanks for review! I think I have taken care of all comments. Re-ran Jenkins, centos builds completed OK, osx is still running but I presume it's going to be success Merged and closing.