If you just noticed a little ‘x’ appear next to your application icons, it is because you are in the mode where you can move icons around. When in this mode, you can also delete applications.

 

Tapping the ‘x’ will bring up a window offing you to delete the app or to not (in case you accidentally clicked on the wrong application ‘x’). If you choose to ‘delete’ the application, it is still on your desktop computer so you can always reinstall. Many apps will loose any info you had entered into that application when you delete though. 

 

After removing an app, you will still be in the mode to move icons or delete other applications. You need to click the button on the front of your iPhone (the Home Button) to return to the normal usage mode.