To what degree should fantasy borrow Earth concepts
I've been developing my own fantasy world for quite a while, and one thing I'm wondering about is specific choices in details you choose to present to your reader. Details that mention something Earth related, which could be as small as a material like cotton or as glaring as a drink with cultural implications like coffee. Examples include: grains, materials, minerals, foods, dances, drinks, phrases, and sayings.
I've heard that to some degree fantasy should operate within Earth logic as to not ostracize the readers from it's own world, but I wonder how true that actually is. I know that this is the tried and true method; plenty of fantasy books have borrowed countless concepts from Earth, (bread being the most common one I can think of) but I wonder if at any point doing so would actually take the reader out of the story.
In my specific case, I'm writing sci-fi and fantasy, where there is multiple worlds bridged together with magic, and to a certain extent I'd like to make certain worlds feel 'alien' whilst still operating within the realm of fantasy. One particular point I've been wondering about is measurements and time; a day on one planet would be longer than a day on another planet, and thus our conventional measurements of time don't seem sufficient.