I'm not sure whether 1274 or 1281 was their best chance of success. In 1274 there were far fewer fortifications and the Japanese only assembled maybe 6,000 warriors at most (against at least 10,000 Mongols). Mongol warfare was totally unique to Japan at the time and there was a real element of shock. It's also clear the Mongols only left because one general was wounded and they believed they only encountered the vanguard of the Japanese force. HOWEVER, in 1274, the Yuan were in the midst of the war with Southern Song, who still had many, many fortifications. A lot of the best Yuan generals and veteran warriors were still involved in that conflict, so it's very possible Japan would've gotten things together and assembled a large enough force (and we know they could, because the systems already existed and they summoned about 60,000 men just 7 years later) that would've dispersed the Mongols. It also seems clear there was no typhoon--the ships sank because they were hurriedly built and even rough weather might cause damage.
1281 was, as noted, when the Yuan had a free hand since there were no ongoing wars besides the whole issue with Kaidu in the west (and those were different troops and generals fighting him for the most part). While there were more fortifications, I think had the two main fleets linked up sooner, or perhaps if instead of waiting, the (mostly) Goryeo fleet which arrived first had taken initiative and done a fullscale attack on a location
beside Hakata instead of mere raids, they probably would've won. Note that this avoids the typhoon since they'd be elsewhere when it passes by. Likewise, attacking Hokkaido and menacing Mutsu Province likely would've done some good too in terms of stirring up internal issues in Japan, diverting armies, and draining the funds of the Shogunate and their vassals. So given the size of the invasion, this one is probably about even to 1274.
1283 is probably too late since they were building more and more castles by that point, although these were more hill fortresses and fortified manors than later era Japanese castles. Morale was also much lower, low enough it provoked a pro-Song rebellion in areas where there was heavy recruitment/shipbuilding. The only positive is that Japan is maybe 1 year from Houjou Tokimune's death--his successor Houjou Sadatoki was less than ideal and about a year after he started his term as regent and head of the Houjou clan by purging a chief minister (Adachi Yasumori), his clan, and anyone deemed too friendly with said clan. Around 500 vassals, many prominent figures, were killed or committed suicide. Worse, this minister was favoured among Shogunal vassals (as opposed to Houjou personal vassals) because he supported policies alleviating the increasing poverty the vassals had worked themselves into, in part because of things like being forced to stand guard duty against the Mongols!
The latter is why the Mongols stand a real fighting chance, since if they gain a good beachhead and win 2-3 more battles in Kyushu, that would get the ball rolling on the collapse of the Kamakura Shogunate. And they easily could--once/if the reinforcement army (in 1281 it was 60K strong) is crushed, then the Shogunate will panic and have to cobble something together
fast. Just like Japan learned about Mongol tactics, the Mongols would adapt to the Japanese style of raiding and delaying. My guess (and slight spoilers for my TL) is that it's very probable Houjou Sadatoki panics and still murders Adachi (since Taira Yoritsuna, Adachi's rival, would still want him dead) and that would cause the OTL purge to become an actual rebellion, far larger than the small rebellion led by Kyushu lord Shouni Kagesuke (whether Shouni actually leads it like my TL depends on whether he survives the invasion, but if not him, I could imagine Kyushu refugees playing a prominent role). The rebellion would have no chance, but it would be the moment you see a sizable number of Japanese, including some noteworthy figures (maybe Shouni if he's still alive, maybe one of the provincial Adachi clan, whose head was Yasumori's second cousin, etc.) defecting to the Mongols.
At that point I feel the Mongols just need a bridgehead to Honshu. Nagato Province was very strongly fortified, and would be even moreso TTL, and the Kanmon Straits have some notorious characteristics (see Battle of Dan-no-ura), but it should be plenty possible assuming Mongol will is still there. And once that bridgehead is established, I think the Yuan could let the anti-Shogunate Japanese do a lot of the heavy lifting. That means burdens like conscripting their soldiers and building ships for the Mongols. On the bright side, because all of Kyushu's land is legally owned by powerful religious institutions and court nobles who will oppose the Yuan, they can buy a lot of loyalty via land reform (although a lot of it will go to their own soldiers). Japan in the late 13th century is fairly fragile between increasingly ineffective, inflexible Shogunate and a succession dispute between two branches of the Imperial family which engulfed the whole Imperial court (which still had some modicum of power and influence in this era) in intrigue that occasionally turned murderous.
Mongol success at this point is determined by how much they're willing to spend on it. It'll really impact the Burma and Vietnam campaigns (the latter already was hurt IOTL by lack of shipping), although
probably won't hurt dealing with Kaidu/Chaghatai Khanate or internal revolts like Prince Nayan's rebellion. Definitely no Java campaign TTL, or it'll be a bare minimum. Kublai Khan was the one most obsessed with invading Japan, so that gives the Mongols until 1294. I think in that time they could take a few provinces in Honshu, maybe conquer Shikoku, but
probably not reach as far as Kyoto.
Of Kublai's successors, Temur Khan was an isolationist, but its possible he could be persuaded that the lesser of two evils is doing one last invasion to finish matters in Japan. In that case, it's possible Kyoto falls and
possibly a total conquest could be achieved. There was also Buyantu Khan (Ayurbarwada) who was an effective ruler and probably would have also wished to finish the conquest of Japan (since he would have been nagged at it by whatever vassal state was established there). After 1320 IOTL, the Yuan fell into a decade of infighting and chaos that was only resolved by Toghon Temur in the 1330s and 1340s, but at that point Toghon also faced famines and epidemics that ultimately ended the Yuan. So in practice, early 1320s will be the time limit, and I think 40 years will be enough for the Yuan to grind their way from Hakata to Kamakura with the aid of plenty of Japanese turncoats.
IIRC the Mongols were withdrawing even without the storms the first time. Conquering Japan would be difficult, as it's mountainous and advanced enough to fight the Mongols, especially as the Japanese reverse engineer Mongol weaponry. And the conquest wouldn't really be worth it. It'd be like European colonialism OTL: Japan is too strong to easily conquer while not being able to pay for itself. Even if they do conquer Japan, they wouldn't be able to subjugate it.
They withdrew in 1274 because they did not expect the tenacity at which the strongly outnumbered Japanese would fight, even at night. Further, one of the generals (Liu Fuheng), was wounded in one of these encounters. There was a dispute within the Mongol leadership--had they pressed the attack, they probably would have won since the state of Japanese fortifications was very poor at the time (fortified manors plus the ruins of 7th century fortresses) and the Shogunate's army was outnumbered.
As for Japan paying for itself, it was widely believed to be a land of gold since it had many productive gold mines. There was some truth to this. Kyushu was also a huge exporter of rice to China, which was paid for by vast imports of copper coins which at least for Southern Song had caused some monetary problems.
I would definitely not call the Kamakura Shogunate an advanced nation compared to Yuan or even Goryeo. Their ships were smaller (but effective at letting their warriors board enemies), their transportation primitive (especially inland), their economy only beginning to be developed. Centuries of relative peace had caused the military to decay in terms of doctrine and weapons (hence why they did so poorly at times in 1274)--they had no crossbows and useful weapons of later times like the yari or the system of mobilising peasants as ashigaru footsoldiers were practically invented for the Mongol campaign. While there will be a tech transfer, it will have to wait quite some time. They'll only get bombs and guns as a result of people re-defecting, and same thing with other tech like advanced trebuchets or other siege weapons.
