
Learn Amazon SageMaker
By :

Before we dive into scaling techniques, let's first discuss the monitoring information that we should consider when deciding whether we need to scale and how we should do it.
Two sources of information are available: the training log and the infrastructure metrics in Amazon CloudWatch.
The training log tells us how long the job lasted. In itself, this isn't really useful. How long is too long? This feels very subjective, doesn't it? Furthermore, even when training on the same dataset and infrastructure, changing a single hyperparameter can significantly alter training time. Batch size is an example, and there are many more.
When we're concerned about training time, I think we're really trying to answer three questions: