Possible Clues:

• Don't mind work with Smith at first
• See 36-Across
• Does Smith not remember to start to work there?
• Did not mind the smithy before tea
• Shut out
• In order to obtain this, fail to mind it
• Neglect fashion as time goes on
• Stop thinking about fashion over time
• You may not recall that's where to find Smith before tea
• Be affected by a recall problem