How do we recognize Elijah when he will come before the day of Lord?

In Matthew 17:11 this could indeed refer to a prophecy present in Malachi 4:5 like you said

Behold, I will send you Elijah the prophet before the coming of the great and dreadful day of the Lord

Reading from Revelation 11:3-6

3 And I will give power unto my two witnesses, and they shall prophesy a thousand two hundred and threescore days, clothed in sackcloth.

4 These are the two olive trees, and the two candlesticks standing before the God of the earth.

5 And if any man will hurt them, fire proceedeth out of their mouth, and devoureth their enemies: and if any man will hurt them, he must in this manner be killed.

6 These have power to shut heaven, that it rain not in the days of their prophecy: and have power over waters to turn them to blood, and to smite the earth with all plagues, as often as they will.

In this Revelation passage, my KJV study bible claims about that specific passage we know these two witnesses can protect themselves prevent rain and perform miracles if necessary, bringing to mind Moses and Elijah who exhibited these types of miracles (Exodus 7:19-20; 1 Kings 17:1). Still, their identity is unknown.

Is there something else we know? Yes. In Matthew 11:14

And if ye will receive it, this is Elias, which was for to come.

and in Matthew 17:13 (two verses right after your reference)

Then the disciples understood that he spake unto them of John the Baptist.

Here we see Jesus identifying John the Baptist as Elijah and the disciples understanding it.

So, when you say

but how do we recognize Elijah when he will come before the Lord?

that's an affirmation that isn't necessarily supported by the Scriptures... We don't know if Elijah is really to come or if that prophecy was already fulfilled in John the Baptist.

How do we recognize Elijah when he will come before the day of Lord?

In the Book of Revelation we read about two witnesses who will be God’s prophets that will prepare the world for Christ’s second coming.

The Two Witnesses

11 I was given a reed like a measuring rod and was told, “Go and measure the temple of God and the altar, with its worshipers. 2 But exclude the outer court; do not measure it, because it has been given to the Gentiles. They will trample on the holy city for 42 months. 3 And I will appoint my two witnesses, and they will prophesy for 1,260 days, clothed in sackcloth.” 4 They are “the two olive trees” and the two lampstands, and “they stand before the Lord of the earth.” 5 If anyone tries to harm them, fire comes from their mouths and devours their enemies. This is how anyone who wants to harm them must die. 6 They have power to shut up the heavens so that it will not rain during the time they are prophesying; and they have power to turn the waters into blood and to strike the earth with every kind of plague as often as they want.

7 Now when they have finished their testimony, the beast that comes up from the Abyss will attack them, and overpower and kill them. 8 Their bodies will lie in the public square of the great city—which is figuratively called Sodom and Egypt—where also their Lord was crucified. 9 For three and a half days some from every people, tribe, language and nation will gaze on their bodies and refuse them burial. 10 The inhabitants of the earth will gloat over them and will celebrate by sending each other gifts, because these two prophets had tormented those who live on the earth.

11 But after the three and a half days the breath[b] of life from God entered them, and they stood on their feet, and terror struck those who saw them. 12 Then they heard a loud voice from heaven saying to them, “Come up here.” And they went up to heaven in a cloud, while their enemies looked on.

13 At that very hour there was a severe earthquake and a tenth of the city collapsed. Seven thousand people were killed in the earthquake, and the survivors were terrified and gave glory to the God of heaven. - Revelation 11:1-13

Who are these two witnesses. Most Christians think it will be Elijah and Enoch. Some think it may be Elijah and Moses.

In the Book of Revelation, the two witnesses are two of God's prophets who are seen by John of Patmos, during the "Second woe" recorded in Revelation 11:1-14. They have been variously identified by theologians as two people, as two groups of people, or as two concepts. Dispensationalist Christians believe that the events described in the Book of Revelation will occur before and during the Second Coming. The two witnesses are never identified in the Christian Bible. Some believe they are Enoch and Elijah, as in the Gospel of Nicodemus, since they are the only two that did not see death as required by the Scriptures. Others believe them to be Moses and Elijah because they appeared during the transfiguration of Jesus, or because Enoch was not Abraham's descendant. Some also believe that they are Moses and Elijah due to the description of what they are to do. They have the power to shut the heavens (Elijah) and turn water into blood (Moses) - Two witnesses

In any case, if Elijah and Enoch do show up there is no sure way that we will be able to recognize him at. The Scriptures simply tells us what to look for, but not the absolute certainty that they are going to be clearly identified by believers at all.

Look at the uncertainty that surrounded the ministry of Jesus himself:

Peter's Confession of Christ

13When Jesus came to the region of Caesarea Philippi, He questioned His disciples: “Who do people say the Son of Man is?” 14 They replied, “Some say John the Baptist; others say Elijah; and still others, Jeremiah or one of the prophets. 15“But what about you?” Jesus asked. “Who do you say I am?”… - Matthew 16:13-15

There will certainly some who will be able to recognize the Two Witness, but how and when will have to be determined as the future events partaking of the Second Coming of Christ unfold!

For myself, it is not an impossibility that Elijah will preach to the Jews and Enoch to those who are neither Christian or Jewish. For Christians, we have our Church leaders to help us prepare for the Great Day of Christ’s Return.

Nothing is written in stone!

Theodoret wrote that Elijah would do just that:

"Paul insists that only a part of Israel has been hardened, for many of them believe. He thus encourages them not to despair that others will be saved as well. After the Gentiles accepted the gospel, the Jews would believe, when the great Elijah would come to them and bring them the doctrine of the faith. The Lord himself said as much: 'Elijah will come and will restore all things.'"