This week I had a warning pop up in one of my R scripts. Usually this warning means that something changed in the underlying data table. Sure enough, one of the fields I was analyzing had a new value that wasn't explicitly mapped in my code. This can happen when a new department, a new survey option, or a new employee type is added, for example.
It got me thinking about different ways to make sure that a data change doesn't affect my automated scripts. In some of my scripts I have a Markdown cell that finds anything unmapped and prints it, but this one I caught because my tidyr::pivot_wider() returned list columns.
My favorite way to catch these is to have a close relationship with the team that makes data changes so that you aren't surprised when they happen. In this case, I had been part of the conversations to change this field, but since I don't use this script as often I forgot to update it right when the change happened. Oops.
We've all been there. At OrganizationView it always seems that the data scientists can develop a new script much faster than the development team can move it to production, often by a significant amount. But the development team don't get caught out by these things. I think we learn a lot from working with them.