Pydantic in 100 Lines (Part 2)
In part 1 of this series we implemented a validator for dataclasses, but we left some important features untouched.
Storing a list in a dataclass is hardly an advaced feature. What about dictionaries, and unions? Are they so hard to implement?
Let’s try, and see for ourselves.
To validate the types of the dataclass fields, we wrote a hydrate_dataclass function, that loops through all the fields of the dataclass, and calls _coerce on each one.
def hydrate_dataclass[T](cls: type[T], data: dict[str, Any]) -> T:
assert is_dataclass(cls)
result = data.copy()
hints = get_type_hints(cls)
for f in fields(cls):
# Skip fields that are not present in the source dictionary.
# When constructing the dataclass, these will become missing
# arguments.
if f.name not in result:
continue
result[f.name] = _coerce(
hints[f.name], result[f.name], f.name
)
unknown = set(data) - {f.name for f in fields(cls)}
if unknown:
raise ValueError(f"Unknown fields: {sorted(unknown)}")
# Build the dataclass from the dictionary and return it
return cls(**result)
Validating Lists
Lists seem like they might be easiest to implement. The call to get_type_hints will tell us that field is a list, we walk through each item, … and voila! We have a list!
But what is the type of the elements in the list?
Let’s start with a new dataclass:
@dataclass
class Person:
name: str
hobbies: list[str]
When we call get_type_hints on it, we get
>>> get_type_hints(Person)
{'name': <class 'str'>, 'hobbies': list[str]}
We know that hobbies is a list, but pulling out the fact that each element in the list is a string is an extra step.
Python, batteries include and all, has the right tool waiting for us: the get_origin, and get_args functions are lying in wait to split list[str] into it’s constituent parts:
>>> get_origin(list[str])
<class 'list'>
>>> get_args(list[str])
(<class 'str'>,)
When we find list field, we need to pull out the type of the list elements, and then cast each each value in the list.
def _coerce[T](
field_type: Type[T], value: Any, field_name: str
) -> T:
# list[str] is not a class, so unpack it into the list type and
# the element type before the isinstance() check below.
if get_origin(field_type) is list:
(element_type,) = get_args(field_type)
if not isinstance(value, list):
err = (
f"Invalid {value!r} for '{field_name}'. "
f"Expected list."
)
raise ValueError(err)
# Tell the checker the coerced list is the field type T.
return cast(
T,
[
_coerce(element_type, element_value, field_name)
for element_value in value
],
)
# If the value has the correct type, move on
if isinstance(value, field_type):
return value
# Otherwise, try to cast it to the correct type
try:
# Tell the checker the class accepts the raw value.
return cast(Callable[[Any], T], field_type)(value)
except (ValueError, TypeError):
err = (
f"Invalid value {value!r} for '{field_name}'. "
f"Expected '{field_type.__name__}'."
)
raise ValueError(err) from None
That was easier than expected. Let’s move on!
Validating Dictionaries
Dictionaries have exactly the same issue as lists. Once we find a dictionary, we need to find the type of the keys, and the type of the values.
Let’s look at an example:
@dataclass
class Person:
name: str
skills: dict[str, int]
With the magic of get_type_hints and our new friends get_origin / get_args, we can see that the dict is composed of
>>> get_type_hints(Person)
{'name': <class 'str'>, 'skills': dict[str, int]}
>>> get_origin(dict[str, int])
<class 'dict'>
>>> get_args(dict[str, int])
(<class 'str'>, <class 'int'>)
The types of the dictionaery key and fields are just the first and second fields returned by get_args.
Emulating what we did for list, we need to add a new special case for dict and cast each key and value to the correct type.
def _coerce[T](
field_type: Type[T], value: Any, field_name: str
) -> T:
if get_origin(field_type) is list:
...
# dict[str, int] is not a class, so unpack it into the key type
# and the value type before the isinstance() check below.
if get_origin(field_type) is dict:
(key_type, value_type) = get_args(field_type)
if not isinstance(value, dict):
err = (
f"Invalid {value!r} for '{field_name}'. "
f"Expected dict."
)
raise ValueError(err)
# Tell the checker the coerced dict is the field type T.
return cast(
T,
{
_coerce(
key_type, key, f"{field_name}<key>"
): _coerce(
value_type, item_value, f"{field_name}[{key!r}]"
)
for key, item_value in value.items()
},
)
# If the value has the correct type, move on
if isinstance(value, field_type):
return value
# Otherwise, try to cast it to the correct type
try:
# Tell the checker the class accepts the raw value.
return cast(Callable[[Any], T], field_type)(value)
except (ValueError, TypeError):
err = (
f"Invalid value {value!r} for '{field_name}'. "
f"Expected '{field_type.__name__}'."
)
raise ValueError(err) from None
Validating Optionals & Unions
Sometimes we want values to be optional. Some people have nicknames, others don’t. We can’t ask everybody to have a nickname just to make our model happy.
@dataclass(frozen=True)
class Person:
name: str
# Alternatively: Optional[str]
nickname: str | None = None
What does get_type_hints say about the type now?
>>> get_type_hints(Person)
{'name': <class 'str'>, 'nickname': str | None}
The type of nickname is str | None. When we take a deeper look with our friends get_origin and get_args, we see
>>> get_origin(str | None)
<class 'types.UnionType'>
>>> get_args(str | None)
(<class 'str'>, <class 'NoneType'>)
That makes it easy. Just like list and dict, if get_origin tells us that the type is a UnionType, then we branch and try to cast to one of the types returned by get_args.
def _coerce(field_type, value, field_name: str):
...
if get_origin(field_type) is UnionType:
union_args = get_args(field_type)
# Go through all the types in the union
for member_type in union_args:
try:
# And try to cast the value to that type
return _coerce(member_type, value, field_name)
except (ValueError, TypeError):
continue
# If none fit, raise an error
raise ValueError(
f"Invalid {value!r} for '{field_name}'. "
f"Expected {field_type}."
)
...
We still have a small bug to fix.
>>> hydrate_dataclass(
... Person, {"name": "Sonny Hill Jr.", "nickname": None}
... )
Person(name='Sonny Hill Jr.', nickname='None')
Since the str annotation can strignify None, the nickname becomes ‘None’ (a str value). To guard against this, we need to explicitly check for None.
def _coerce[T](
field_type: Type[T], value: Any, field_name: str
) -> T:
if get_origin(field_type) is list:
...
if get_origin(field_type) is dict:
...
# str | None is not a class, so unpack it into its member types
# before the isinstance() check below.
if get_origin(field_type) is UnionType:
union_args = get_args(field_type)
# If the value is None, but the union does not explicitly support
# None, then we are missing a value in the dataclass.
if value is None:
if type(None) in union_args:
return cast(T, None)
err = (
f"Invalid None for '{field_name}'. "
f"Expected {field_type}."
)
raise ValueError(err)
# Try the non-None members of the union in order, and keep
# the first type that the value can be cast to.
for member_type in union_args:
if member_type is type(None):
continue
try:
return _coerce(member_type, value, field_name)
except (ValueError, TypeError):
continue
# If none fit, raise an error
err = (
f"Invalid {value!r} for '{field_name}'. "
f"Expected {field_type}."
)
raise ValueError(err)
# If the value has the correct type, move on
if isinstance(value, field_type):
return value
# Otherwise, try to cast it to the correct type
try:
# Tell the checker the class accepts the raw value.
return cast(Callable[[Any], T], field_type)(value)
except (ValueError, TypeError):
err = (
f"Invalid value {value!r} for '{field_name}'. "
f"Expected '{field_type.__name__}'."
)
raise ValueError(err) from None
Putting it all together
An underappreciated aspect of Python is how little magic the language has. Even core concepts such as classes are implemented using basic language primitives. This can make it surprisingly easy to introspect the language at runtime.
To me, this makes the language fascinating, you can always pull back the veil on language features to discover how they work.