My analysis, based on Archive.org:
Methodology: last visit of Archive.org crawler on Twitter.com/KanyeWest
https:// web.archive.org/web/*/https:// twitter.com/KanyeWest
April
5 : >404
6 : >404
9 : >404
10 : >404
11 : >404
14 : 856K followers, 7 tweets, 1 following, 1 like
15 : 5.24M followers, 8 tweets, 1 following, 1 like
16 : 6.96M followers, 14 tweets, 1 following, 1 like
17 : 8.38M followers, 12 tweets, 1 following, 1 like
18 : 13.6M followers, 41 tweets, 1 following, 1 like
19 : 14.4M followers, 43 tweets, 1 following, 1 like
21 : 21.1M followers, 87 tweets, 1 following, 1 like
22 : 24.5M followers, 108 tweets, 1 following, 1 like
23 : 27.2M followers, 130 tweets, 1 following, 1 like
24 : 27.8M followers, 139 tweets, 1 following, 1 like
25 : 27.9M followers, 208 tweets, 1 following, 1 like
live (see post time, not archived by Archive.org yet): 19M followers, 220 tweets, 1 following, 1 like
So yes, he lost a lot of followers, but somehow I wouldn't believe he lost them without some Twitter admin support. Apparently the growing number is due to account reactivation.